11 research outputs found
Coplanar Repeats by Energy Minimization
This paper proposes an automated method to detect, group and rectify
arbitrarily-arranged coplanar repeated elements via energy minimization. The
proposed energy functional combines several features that model how planes with
coplanar repeats are projected into images and captures global interactions
between different coplanar repeat groups and scene planes. An inference
framework based on a recent variant of -expansion is described and fast
convergence is demonstrated. We compare the proposed method to two widely-used
geometric multi-model fitting methods using a new dataset of annotated images
containing multiple scene planes with coplanar repeats in varied arrangements.
The evaluation shows a significant improvement in the accuracy of
rectifications computed from coplanar repeats detected with the proposed method
versus those detected with the baseline methods.Comment: 14 pages with supplemental materials attache
The World of Fast Moving Objects
The notion of a Fast Moving Object (FMO), i.e. an object that moves over a
distance exceeding its size within the exposure time, is introduced. FMOs may,
and typically do, rotate with high angular speed. FMOs are very common in
sports videos, but are not rare elsewhere. In a single frame, such objects are
often barely visible and appear as semi-transparent streaks.
A method for the detection and tracking of FMOs is proposed. The method
consists of three distinct algorithms, which form an efficient localization
pipeline that operates successfully in a broad range of conditions. We show
that it is possible to recover the appearance of the object and its axis of
rotation, despite its blurred appearance. The proposed method is evaluated on a
new annotated dataset. The results show that existing trackers are inadequate
for the problem of FMO localization and a new approach is required. Two
applications of localization, temporal super-resolution and highlighting, are
presented
Sub-frame Appearance and 6D Pose Estimation of Fast Moving Objects
We propose a novel method that tracks fast moving objects, mainly non-uniform
spherical, in full 6 degrees of freedom, estimating simultaneously their 3D
motion trajectory, 3D pose and object appearance changes with a time step that
is a fraction of the video frame exposure time. The sub-frame object
localization and appearance estimation allows realistic temporal
super-resolution and precise shape estimation. The method, called TbD-3D
(Tracking by Deblatting in 3D) relies on a novel reconstruction algorithm which
solves a piece-wise deblurring and matting problem. The 3D rotation is
estimated by minimizing the reprojection error. As a second contribution, we
present a new challenging dataset with fast moving objects that change their
appearance and distance to the camera. High speed camera recordings with zero
lag between frame exposures were used to generate videos with different frame
rates annotated with ground-truth trajectory and pose
Learned Semantic Multi-Sensor Depth Map Fusion
Volumetric depth map fusion based on truncated signed distance functions has
become a standard method and is used in many 3D reconstruction pipelines. In
this paper, we are generalizing this classic method in multiple ways: 1)
Semantics: Semantic information enriches the scene representation and is
incorporated into the fusion process. 2) Multi-Sensor: Depth information can
originate from different sensors or algorithms with very different noise and
outlier statistics which are considered during data fusion. 3) Scene denoising
and completion: Sensors can fail to recover depth for certain materials and
light conditions, or data is missing due to occlusions. Our method denoises the
geometry, closes holes and computes a watertight surface for every semantic
class. 4) Learning: We propose a neural network reconstruction method that
unifies all these properties within a single powerful framework. Our method
learns sensor or algorithm properties jointly with semantic depth fusion and
scene completion and can also be used as an expert system, e.g. to unify the
strengths of various photometric stereo algorithms. Our approach is the first
to unify all these properties. Experimental evaluations on both synthetic and
real data sets demonstrate clear improvements.Comment: 11 pages, 7 figures, 2 tables, accepted for the 2nd Workshop on 3D
Reconstruction in the Wild (3DRW2019) in conjunction with ICCV201
Tracking by 3D Model Estimation of Unknown Objects in Videos
Most model-free visual object tracking methods formulate the tracking task as
object location estimation given by a 2D segmentation or a bounding box in each
video frame. We argue that this representation is limited and instead propose
to guide and improve 2D tracking with an explicit object representation, namely
the textured 3D shape and 6DoF pose in each video frame. Our representation
tackles a complex long-term dense correspondence problem between all 3D points
on the object for all video frames, including frames where some points are
invisible. To achieve that, the estimation is driven by re-rendering the input
video frames as well as possible through differentiable rendering, which has
not been used for tracking before. The proposed optimization minimizes a novel
loss function to estimate the best 3D shape, texture, and 6DoF pose. We improve
the state-of-the-art in 2D segmentation tracking on three different datasets
with mostly rigid objects
Human from Blur: Human Pose Tracking from Blurry Images
We propose a method to estimate 3D human poses from substantially blurred
images. The key idea is to tackle the inverse problem of image deblurring by
modeling the forward problem with a 3D human model, a texture map, and a
sequence of poses to describe human motion. The blurring process is then
modeled by a temporal image aggregation step. Using a differentiable renderer,
we can solve the inverse problem by backpropagating the pixel-wise reprojection
error to recover the best human motion representation that explains a single or
multiple input images. Since the image reconstruction loss alone is
insufficient, we present additional regularization terms. To the best of our
knowledge, we present the first method to tackle this problem. Our method
consistently outperforms other methods on significantly blurry inputs since
they lack one or multiple key functionalities that our method unifies, i.e.
image deblurring with sub-frame accuracy and explicit 3D modeling of non-rigid
human motion.Comment: typos and minor error fixe
Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects
We address the novel task of jointly reconstructing the 3D shape, texture,
and motion of an object from a single motion-blurred image. While previous
approaches address the deblurring problem only in the 2D image domain, our
proposed rigorous modeling of all object properties in the 3D domain enables
the correct description of arbitrary object motion. This leads to significantly
better image decomposition and sharper deblurring results. We model the
observed appearance of a motion-blurred object as a combination of the
background and a 3D object with constant translation and rotation. Our method
minimizes a loss on reconstructing the input image via differentiable rendering
with suitable regularizers. This enables estimating the textured 3D mesh of the
blurred object with high fidelity. Our method substantially outperforms
competing approaches on several benchmarks for fast moving objects deblurring.
Qualitative results show that the reconstructed 3D mesh generates high-quality
temporal super-resolution and novel views of the deblurred object.Comment: Accepted to 35th Conference on Neural Information Processing Systems
(NeurIPS 2021